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EXECUTIVE SUMMARY 


The purpose of this audit was to determine what caused the problems associated with the 
Electronic Document Management System (EDMS) Initiative and to develop recommendations 
that the Office of Information Systems could use to improve their processes. This report 
communicates the final audit results of the Management of the EDMS Initiative - Contract 
P0162. Each audit objective focused on a different, key phase of the EDMS initiative. A 
previous interim report dated March 31, 2005, addressed the findings associated with Objectives 
1 and 2 which have been included as Appendix A of this report. This report addresses 
Objectives 3 and 4. The four audit objectives were: 





= Objective 1 — Did the project goals, procedures, specifications, and management 
direction effectively ensure the realization of EDMS goals as stated in Policy 
Statement 000-325-008? 

= Objective 2 — Did the vendor selection process for the EDMS vendor integrator fully 
comply with established procedures and efficiently and effectively ensure an 
appropriate vendor selection? 

= Objective 3 — Did the contract negotiation process fully comply with established 
procedures and efficiently and effectively ensure that contract terms and fees were 
fairly negotiated? 

= Objective 4 — Did contract and project management fully comply with established 
procedures and result in efficiently and effectively implemented EDMS applications? 


The contract negotiation process did not fully comply with established procedures and failed to 
effectively ensure that contract terms and fees were fairly negotiated. Additionally, the contract 
and project management of Contract P0162 did not fully comply with established procedures and 
failed to result in efficiently and effectively implemented EDMS applications. 


To address contract negotiation issues in place for term contracts procured in the Office of 
Information Systems (OIS) we recommend the Assistant Secretary for Finance and 
Administration require that: 


= Personnel responsible for contracting on behalf of the Department, including Project 
Managers, are adequately trained to perform all functions associated with 
procurement and acquisition. Training should include participation in the Department 
of Management Services’ Contractor Negotiation Certification Training. 
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Personnel granted signatory authority are trained in both ethics and the 
responsibilities associated with the authority granted them. These individuals should 
sign annual conflicts of interest statements identifying all potential conflicts, both 
actual and perceived. Review of these statements should be made part of each 
acquisition or procurement, and should be included in the contract files. 

Contracts are clearly written, technically sufficient, and supportive of project 
performance monitoring. 

Contract negotiations adhere to the Procurement Office’s Negotiations Handbook 
published January 28, 2005. 

The Procurement Office review contract negotiation procedures used to procure other 
professional services to develop contract negotiation procedures for IT related 
services. Contract negotiations for IT related services should include: 

o Thoroughly documented development of labor rates, hours to be expended, 
direct expense details, overhead and profit margin development, and other 
aspects of cost and price analysis; 

Documentation of planning; 

Bargaining characterized by the skillful use of a variety of negotiation tactics; 

Thorough market research; 

Analysis of price proposals based on the current market situation and specific 

contract requirements; 

o Use of a knowledgeable negotiating team and the involvement of 
management; 

o Certification of contractor’s proposed costs; and 

o Documentation of all analyses and decisions based on those analyses. 


O00 0 


To strengthen OIS contract and project management processes, we recommend that the Assistant 
Secretary for Finance and Administration require: 
= The Business System Support Office Manager to develop, implement, enforce, and 
monitor project management procedures that includes current project management 
practices and emphasizes the importance of ethics and fiscal responsibility. These 
procedures should require Project Managers to: 


o Assess the reasonableness of invoices in relation to actual project progress; 
o Review invoices for statutory and procedural compliance; and 
o Document that all goods and services received conform to requirements and 
specifications established in the contract, base line documents, and/or task 
assignments. This may be accomplished by adding a signed statement to the front 
of the Purchase Order Delivery Notice when submitting an invoice for payment. 
The Chief Information Officer to better support OIS’ project managers by assigning 
an OIS employee, as envisioned by the ISDM, to serve as contract manager for Type 
B (Contractual/Miscellaneous) and other atypical, higher-risk contracts; 
The Chief Information Officer to incorporate standardized and comprehensive 
contract and project management training into OIS’ current training program. This 
training should emphasize ethics and responsibility and include elements of the 
Department’s Project Management Training course that apply to the management of 
Type B (Contractual/Miscellaneous) contracts. 
The Chief Information Officer require Information Resource Requests and Purchase 
Orders to include the deliverables to be received, establish the required date of 
completion, hours to be expended, billing rates, and costs to be reimbursed when the 
primary procurement document (e.g., contract, task assignment, etc...) does not 
specifically these elements. 
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BACKGROUND AND INTRODUCTION 





This audit was performed by the Office of Inspector General as a result of contract and project 
management weaknesses identified in our Contract Audit No. 140-2014; Yang Enterprises, Inc., 
(YED Contracts BA974 and P0162, dated March 29, 2004. The Department is engaged in legal 
proceedings pertaining to YEI and the former EDMS Project Manager, Ms. Mavis Georgalis. 


Finding 3 — Contract Negotiation Process 

YEI participated in two separate negotiation phases; the first negotiation established the fees 
associated with the pilot project; the second negotiation established the fees associated with 
Contract P0162. 


Pilot Project - 

In March of 1998, as part of the Department’s EDMS vendor selection process, Yang 
Enterprises, Inc. (YEI) and Blueridge Technologies, Inc. were asked to submit price proposals as 
a prerequisite to participating in a Live Test Demonstration. According to the Invitation To 
Negotiate (ITN), these price proposals were required to be scored and ranked to determine which 
vendor would be asked to proceed with the pilot project. ITN Document #JA0698D1, Section 
7.5 — Negotiation, required each selected vendor to participate in the Live Test Demonstration 
submit résumés for personnel who may be used to provide professional services, the rates for 
those services, and a breakdown of how the rates were calculated. No résumés or a breakdown 
of the proposed costs were located. Although price proposals were submitted, no evidence was 
found indicating that the price proposals were scored and ranked. YEI’s proposed rates were 
greater than those proposed by Blueridge Technologies, Inc. 


After YEI was chosen to perform the pilot project, negotiations were conducted to develop the 
rates and fees to be billed to the Department for YEI’s pilot project work. The rates and fees 
negotiated for pilot project billing were based upon the rates and fees included in YEI’s initial 
price proposal. Former EDMS Project Manager, Mavis Georgalis, and procurement office staff 
participated in this process. 


In early July of 1998, Mavis Georgalis informed the Chief Information Officer that the labor 
billing rates to be paid by the Department to YEI for professional services relating to the 
Construction Document Management System (CDMS) pilot project were to be based on the mid- 
range of FDOT salary structures for required employee classifications (e.g. Data Processor I) 
plus 158% overhead (the maximum overhead allowed as of October 1, 1995) and 10% profit 
margin. 
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The Department did not pay these rates. In a letter from YEI, dated July 24, 1998, YEI 
confirmed the agreed upon rates in connection with the pilot project with former EDMS Project 
Manager Georgalis. Ms. Georgalis signed the agreement indicating that she agreed to the 
following terms and conditions relating to the pilot project: 























CLASSIFICATION RATE 
Project Manager $150 
Senior System Analyst $ 76 
Trainer $116 
Documentation Specialist $ 60 
Technical Support Specialist $150 











For all classifications, except the senior systems analyst, these rates were substantially higher 
than the rates calculated by burdening the mid-range of the Department’s salary structures and 
were not supported by résumés or include an explanation of how the rates were developed. 


Contract Rates - 

In January of 1999, after YEI was awarded the EDMS contract, the Department negotiated with 
YEI to establish contract rates and fees. The Department established a budgetary ceiling of 
$8,000,000 based on recommendations made in Universal Systems, Inc.’s (USI) Functional 
Specifications Document. Labor billing rates for Contract P0162 were developed based on the 
mid-range of Department career service employee classifications related to data processing plus 
160% overhead and a 10% profit margin, with the exception of the Project Manager and 
Technical Support Specialist classifications, which were set at the highest Departmental rate. 
Documentation justifying the use of the high point for these two classifications was not located 
in project files. No rationale for using the overhead rate of 160% or profit margin of 10% was 
noted. No support was found to indicate whether YEI’s overhead was similar to or greater than 
these rates. 


Labor billing rates were significantly lower than both the rates initially proposed by YEI and the 
rates used to bill for pilot project effort, which indicates that the rates initially proposed by YEI 
and billed to FDOT during the pilot project were inflated. 


For the purposes of analyzing the reasonableness of YEI’s labor billing rates, we compared 
YEI’s rates to those of ICF Kaiser and Montalbano and Associates. YEI’s labor rates generally 
exceed ICF Kaiser’s labor billing rates for work performed under Contract C5767 for similar 
labor classifications. For Contract C5767, labor rates were developed in a manner similar to the 
method used to develop Contract P0162 rates. 


YEP s labor billing rates exceeded those used by Montalbano and Associates to bill for EDMS 
work on Consultant Commodity Contract 975-556-00-1. A former YEI employee eventually 
went to work for Montalbano and Associates. This former YEI employee continued to provide 
EDMS services similar to those he performed for YEI on Contract P0162. Under Contract 
P0162, YEI billed for this employee’s time at a rate of $79.03. Under Contract 975-556-00-1, 
Montalbano and Associates billed for the same employee at a rate of $45. 


YEP s labor billing rates for Contract P0162 were generally similar to or less than the rates billed 
to the Department for work they performed in connection with Contract C5756 during the 1995- 
1996 fiscal year. Contract C5767 rates were burdened with an overhead rate of 135% and profit 
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margin of 15%, while Contract P0162 rates were burdened with an overhead rate of 160% and 
profit margin of 10%. 


Finding 4 — Contract and Project Management 

The EDMS Manual and Topic 375-055-015, Electronic Document Management would have 
been the sole source of procedures relating to the EDMS Project. However, these procedures 
were never implemented and no other source of EDMS related procedures applied to Contract 
P0162. 


The Specialized Technology Office was created to serve the Department’s EDMS and GIS 


needs. The Chief Information Officer did not direct the Manager of Specialized Technologies, 
Mavis Georgalis, to establish procedures for the Specialized Technology Office. 
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OBJECTIVES, SCOPE, AND METHODOLOGY 





The objectives covered in this Audit Report were: 

Objective 3 - Did the contract negotiation process fully comply with established 
procedures and efficiently and effectively ensure that contract terms and fees were 
fairly negotiated? 

Objective 4 - Did contract and project management fully comply with established 
procedures and result in efficiently and effectively implemented EDMS applications? 


The scope of this audit covers information related to the EDMS initiative beginning in 1992 with 
the Production Management Business Area’s analysis of EDMS through the cancellation of 
Contract P0162 in 2004. Fieldwork began in August of 2004 and was completed in March of 
2005. Former EDMS Project Manager Georgalis was not interviewed due to ongoing litigation. 
We did not assess or draw conclusions regarding the overall success of the EDMS initiative. 


The methodology used to achieve our audit objectives included: 

Review of the EDMS contract and project documentation; 

Analysis of planned verses actual schedules and resources; 

Comparison of contract negotiations processes and contract and project management 
to requirements; 

Review of applicable statutes and procedures; 

Review and analysis of the following information: 


O 


oo0o0o0oọ 


O 


Purchase Orders; 

Invoices; 

Information Resource Requests; 

EDMS Project Schedules; 

Specialized Technology Status Reports; 
YEI Status Reports; and 

YEI Estimates. 


Best practices research; and 
Staff interviews. 


Audit Report No. 330-4002 e Page 6 


Florida Department of Transportation 
Office of Inspector General 


FINDINGS AND RECOMMENDATIONS 








Audit Finding 3 — Contract Negotiation Process 


Objective 


Conclusion 


Supporting 
Evidence 


Did the contract negotiation process fully comply with established 
procedures and efficiently and effectively ensure that contract terms 
and fees were fairly negotiated? 


The contract negotiation process did not fully comply with established 
procedures and failed to effectively ensure that contract terms and fees 
were fairly negotiated. 


It is important to note that OIS infrequently procures services using 
state term contracts like Contract P0162. The unusual nature of 
Contract P0162 indicates that the weaknesses associated with it are 
probably specific to P0162 and do not apply to the majority of 
contracts entered into by OIS. 


In general, OIS is able to successfully negotiate the majority of its 
contracts without difficulties similar to those encountered with 
Contract P0162. P0162 presented several challenges that OIS does 
not encounter on a regular basis, including: new procurement 
methodology, new Project Manager, new technology, reduced field of 
responsive vendors, and an unusual negotiation methodology. 


OIS is also able to successfully negotiate prices in part because OIS 
frequently contracts with smaller, less expensive firms. Since the 
vendor employee works along side of the FDOT personnel and 
usually the task Project Manager, staff augmentation contracts also 
enable an increased level of oversight. 


While our overall conclusion is that the contract negotiation process 
did not effectively ensure that contract terms and fees were fairly 
negotiated, the process was well executed in a number of respects: 
= Negotiations appear to have been timely; 
= Negotiations for the pilot project involved a cross-functional 
team; 
= The ITN identified deal breakers that vendors were required to 
conform to if they were to be considered for the Live Test 
Demonstration; 
= The Live Test Demonstration provided an opportunity for OIS 
to verify vendor claims; 
= The Department developed an overall pricing objective of 
$8,000,000 based on recommendations made in Universal 
Systems, Inc.’s (USI) Functional Specifications Document; 
and 
= The use of an ITN gave the Department additional bargaining 
leverage. 
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However, the following conditions are the basis for our conclusion 
that the contract negotiation process did not fully comply with 
procedures and failed to effectively ensure that contract terms and fees 
were fairly negotiated. Our conclusion is based on our review of the 
contract, procedures, negotiations, and the control environment. 


Review of Contract P0162 revealed weaknesses indicative of the 
failure of the contract negotiations process, specifically: 
= Contract negotiations did not result in a clearly-written, 
technically sufficient contract. 

o Contract language did not facilitate performance 
monitoring or mitigate the risk of loss resulting from 
changes (i.e., Department requirements, vendor’s 
ability to perform, or changes in technology). 

o Contract language did not specifically define 
thresholds for vendor performance, maintenance and 
support response times, non-performance penalties, or 
performance incentives. 

= Contract P0162 did not clearly state specific deliverables. 


The contract negotiations process did not fully comply with the 
contract or existing procedures, specifically: 
= Contract negotiations were performed by former EDMS 
Project Manager, Mavis Georgalis, and did not include at least 
three individuals as required by Section 287.057, F.S., 
Procurement of Commodities or Contractual Services, DMS 
Memorandum #21, and Department Procedure 375-040-020, 
Procurement of Commodities and Contractual Services; and 
= Mavis Georgalis did not issue task assignments detailing 
deliverables, scope of services, limiting amounts, or other 
contractual requirements as prescribed by Section 287.057, 
F.S., Contract P0162, or Department procedures. 


The contract negotiations process failed to incorporate basic elements 
of negotiation, analysis, or verification, specifically: 
= Contract negotiations for Contract P0162 were not well 
documented. 
= YETI did not provide résumés to support employee 
qualifications and YEI’s employee qualifications were not 
evaluated as required by the ITN Document #JA0698D1. 
= Project files contained no evidence that Mavis Georgalis 
reviewed or analyzed indirect costs, overhead, or operating 
margin. Instead, overhead of 160% and profit margin of 10% 
were used to develop Contract P0162 labor rates. 
= Contract and project documentation did not include evidence 
that Mavis Georgalis verified costs, evaluated costs for 
reasonableness and allowability, or compared proposed costs 
with: 
o Actual costs previously incurred by YEI; 
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o Independent cost estimates; 
Forecasts or planned expenditures; 

o Prior estimates or historical data for same of similar 
items; or 

o Cost trends for the same or similar items. 

= The project files did not contain evidence that labor rates or 
estimated labor hours were reviewed for the initial price 
proposal, pilot project proposal, or the contract pricing 
proposal. 

= Contract Negotiations did not incorporate the following best 
practices: 

o Evaluation of the methodology used by YEI to estimate 
costs; 

o Review of cost elements by a multifunctional team; 

o Development of a negotiation plan; or 

o Involvement of senior management. 

= When negotiating pricing, the former EDMS Project Manager, 
Mavis Georgalis, did not adequately use key bargaining 
strategies, specifically: 

o The Department unnecessarily sacrificed bargaining 
power by including budget ceiling in the ITN 
document; and 

o The Department did not utilize bargaining leverage 
afforded by the ITN process when negotiating contract 
pricing. 


O 


Issues with the contract negotiations process are symptomatic of 
weaknesses in the control environment, specifically: 
= The process for granting signatory authority does not 
adequately emphasize ethics and the fiscal responsibilities 
associated with such authority; 
= Conflicts of interest were not identified; and 
= The Department does not adequately train personnel in the 
negotiation and administration of Type B Contracts 
(Contractual/Miscellaneous). 


Section 287.057, Florida Statutes, Procurement of Commodities 
and Contractual Services 

This statute contains procurement requirements, including the 
requirement that a negotiation team consist of at least three employees 
who have experience and knowledge in the program areas and service 
requirements for which contractual services are sought. 


Department of Management Services Memorandum #21 
This memorandum provides ITN process guidelines, including basic 
methodologies for competitive negotiations. 


Procurement Procedure 375-040-020, Procurement of 
Commodities and Contractual Services 
This procedure requires the Contractual Services Unit to maintain a 
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contract file which contains all pertinent documentation of the 
procurement, contracting, and the administrative phases. It also 
requires that the vendor selection process comply with Section 
287.057, F.S., and that three or more people perform the selection. 
This procedure also describes the procurement process as the 
preliminary portion of the overall acquisition process, which includes 
the development of the scope of services, the preliminary 
development of payment methods/terms and conditions, the selection 
of the contractor, and the establishment of pricing information. 


Functional Specifications Document 

Policy 000-325-008 established the Functional Specifications 
Document as the Department’s five-year guide and EDMS 
implementation plan. This document analyzed the potential costs 
associated with developing an Enterprise-wide EDMS. This analysis 
formed management’s basis for establishing an $8,000,000 budgetary 
ceiling. 


Best Practice Research 
Florida Information Technology Consultant Services Audit #2000-06, 
Defense Procurement and Acquisition Policy — Contract Pricing 
Reference Guide, Volumes 3 and 5; US Securities and Exchange 
Commission, Audit No. 350, Administration of Information 
Technology Contracts; FDOT Topic No. 375-030-010-d, 
Amendments and Task Work Orders for Professional Services; and a 
variety of other literature provided common characteristics of 
successful contract negotiations. Based on this research, the 
following are characteristic of successful contract negotiations: 
= Clearly stated deliverables; 
= Cost analysis that includes tests for reasonableness and 
verification and evaluation of costs; 
= Evaluation of the methodology used by the vendor to develop 
proposed cost elements; 
= Participation of a multi-functional team; 
= Development of a negotiation plan; 
= Comparison of proposed costs with: 
o Actual costs previously incurred by the vendor; 
Previous cost estimates proposed by the vendor; 
Other cost estimates received; 
Independent cost estimates; 
Forecasts or planned expenditures; and 
Market information; 
= Use of a variety of bargaining strategies; 
= Adequate training; and 
= Emphasis on ethics. 


00000 


Cause The following factors contributed to the failure of the contract 
negotiation process: 
= Failure of the former EDMS Project Manager, Mavis 
Georgalis, to negotiate contract terms in a manner that 
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promoted the Department’s interests; 

Management did not establish a control structure to ensure 
adequate contract and project management oversight. 
Senior management’s over-reliance on Mavis Georgalis; 
and 

Weaknesses in the control environment related to signatory 
authority, conflicts of interest, contract negotiation and 
administration, and procedures. 


Failure to adequately negotiate contract terms and fees increases the 
risk that the Department will be subject to unfair or weak contractual 
terms and conditions. This condition may result in the Department 
paying too much for goods and services. 


To address contract negotiation issues in place for term contracts 
procured in the OIS we recommend the Assistant Secretary for 
Finance and Administration require that: 


Personnel responsible for contracting on behalf of the 
Department, including Project Managers, are adequately 
trained to perform all functions associated with 
procurement and acquisition. Training should include 
participation in the Department of Management Services’ 
Contractor Negotiation Certification Training. 

Personnel granted signatory authority are trained in both 
ethics and the responsibilities associated with the authority 
granted them. These individuals should sign annual 
conflicts of interest statements identifying all potential 
conflicts, both actual and perceived. Review of these 
statements should be made part of each acquisition or 
procurement, and should be included in the contract files. 
All contracts are clearly written, technically sufficient, and 
supportive of project performance monitoring. 

All contract negotiations adhere to the Procurement 
Office’s Negotiations Handbook published January 28, 
2005. 

The Procurement Office review contract negotiation 
procedures used to procure other professional services to 
develop contract negotiation procedures for IT related 
services. Contract negotiations for IT related services 
should include: 

o Thoroughly documented development of labor 
rates, hours to be expended, direct expense details, 
overhead and profit margin development, and other 
aspects of cost and price analysis; 

o Documentation of planning; 

o Bargaining characterized by the skillful use of a 
variety of negotiation tactics; 

o Thorough market research; 

o Analysis of price proposals based on the current 
market situation and specific contract requirements; 
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o Use of a knowledgeable negotiating team and the 
involvement of management; 
o Certification of contractor’s proposed costs; and 
o Documentation of all analyses and decisions based 
on those analyses. 
Management 
Response 
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Audit Finding 4 — Contract and Project Manag 





Objective 


Conclusion 


Supporting 
Evidence 


Did contract and project management fully comply with established 
procedures and result in efficiently and effectively implemented 
EDMS applications? 


The contract and project management of Contract P0162 did not 
fully comply with established procedures and failed to result in 
efficiently and effectively implemented EDMS applications. 


Use of a term contract significantly impacted the contract and project 
management of the EDMS initiative. Term contracts require a high 
level of skill and vigilance on the part of the project manager and can 
be complicated to manage financially. The former EDMS Project 
Manager, Mavis Georgalis, did not exert the oversight and guidance 
necessary for this project to be successful. Mavis Georgalis did not 
develop and implement adequate procedures, enforce contract terms, 
monitor vendor performance, or inform management of deficiencies 
in vendor performance. Mavis Georgalis approved payment of 
vendor invoices that lacked adequate support. 


As discussed in Finding #2 of the interim report, the selection of 
YEI, as the EDMS vendor integrator, compounded the project 
management issues identified in this finding. 


It is important to note that the management of current OIS contracts 
differs significantly from the way in which Contract P0162 was 
managed. OIS has implemented several controls to ensure that 
future contracts are successful, specifically: 
= Reassigned the EDMS function under the Business System 
Support Office (BSSO) which is subject to the Information 
Systems Development Methodology (ISDM). 
= BSSO’s current practice is to compare actual resources 
expended to resources budgeted and compare hours billed by 
consultants to actual hours worked. 
= The ISDM requires the Contract Manager to ensure that 
invoices are compliant with contract terms and provisions. 
= BSSO incorporates the following best practices into its 
project management methodology: 
o Performing risk assessment during the identification 
and assessment phase of application development; 
o Developing well-defined requirements; 
o Using Microsoft Project to record task information; 
o Conducting project wrap-up/post project summaries 
to discuss ways that future projects could be 
improved; and 
o Assessing project success at several intervals 
throughout the project’s lifecycle. 
= BSSO requires several layers of system and user testing 
against the application requirements. User acceptance is the 
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final layer of testing, and ensures that the end user is 
satisfied with the application. 


The following conditions are the basis for our conclusion that the 
contract and project management of Contract P0162 did not fully 
comply with established procedures and failed to result in efficiently 
and effectively implemented EDMS applications. Our conclusion is 
based on our review of procedures, contract and project 
documentation and processes, financial documentation, and Contract 
P0162. 


PROCEDURES 

Although the EDMS Project is currently subject to the ISDM, review 
of procedures indicates the EDMS Project suffered from a lack of 
procedures. The primary sources of EDMS guidance, the EDMS 
Manual and Topic 325-055-015, Electronic Document Management, 
were never implemented. 


CONTRACT AND PROJECT MANAGEMENT 
DOCUMENTATION 
Inadequate and incomplete contract and project documentation are 
symptomatic of a lack of transparency. Lack of transparency belies a 
lack of accountability. Review of the contract and project files 
revealed inadequate and/or incomplete project documentation, 
specifically: 
= Although Joint Application Development sessions were held, 
documentation describing the content and results of the 
sessions was not available. 
= Official acceptance and sign-off of functional and technical 
specifications was not readily apparent. 
= Application development consistently exceeded estimated 
work effort, but no justification or explanation was in 
evidence. 
= Project schedules were incomplete and sometimes conflicted 
with other documentation. 
= Application testing was conducted, but the testing process 
and results were not sufficiently documented. 
= Deployment and training constituted a significant portion of 
project efforts, but neither the issues that arose during 
deployment and training nor the steps taken to address those 
issues were well documented. 
= Help desk calls were logged, but it was difficult to determine 
how and when those issues were resolved. 
= Overall project progress was difficult to determine. Project 
documentation did not provide the percentage of completion 
as applications were developed or track the degree to which 
the project achieved its goals. 
= Task assignments, as required by the contract, were not 
issued. 
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Vendor estimates were not sufficiently detailed to allow for 
project tracking. 

Application acceptance was not formally documented. 
Instead, issues were logged, discussed, and resolved through 
informal sources. 


CONTRACT AND PROJECT MANAGEMENT PROCESS 
We reviewed project management documentation including 
estimates prepared by the vendor, status reports, project schedules, 
and other tracking documentation to determine the degree to which 
the former EDMS Project Manager, Mavis Georgalis, monitored 
project progress. Mavis Georgalis failed to adequately perform 
project management duties, specifically: 


The project schedule created by Mavis Georgalis was 
abandoned in July of 2000. 

The project schedule did not allow for comparisons of actual 
and expected results. 

Work performed on the project did not adhere to the project 
schedule. 

The Specialized Technology Status Reports submitted to the 
Chief Information Officer did not identify tasks that 
significantly exceeded estimated completion dates. 

The Specialized Technology Status Reports did not contain 
percentage of completion estimates for each application. 
Status report completion dates were rarely met causing 
planned applications to be postponed indefinitely. 

YEI Status Reports did not include revised dates of 
completion. 

YEI did not submit Status Reports during the pilot project or 
during portions of the project after the contract was awarded. 
YEI did not provide estimates of the hours, labor costs, or 
expenses for all tasks or applications. The few estimates that 
YEI submitted did not provide consistent or useful 
information. 

Work on the project significantly exceeded YEI estimates of 
hours, labor costs, and expenses. 

Mavis Georgalis did not hold the vendor to estimates and 
require justifications for deviating from initial estimates. 
YET s estimates were used to develop purchase order limiting 
amounts. As actual work exceeded estimates, new purchase 
orders were issued. 

Mavis Georgalis did not effectively use purchase orders or 
invoices as tools to track project progress. 


FINANCIAL DOCUMENTATION 

Review of purchase orders, invoices, information resource requests, 
and other financial documentation revealed that Mavis Georgalis 
failed to adequately perform basic contract management duties, 
specifically: 


Purchase orders did not cover the expected time to complete 
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a specific task; instead they covered entire fiscal years and 
rarely expressed concrete dates of completion for 
deliverables. 
Purchase orders often expressed deliverables in terms of 
services to be provided, but did not indicate the application 
upon which the work was performed. 
Some of the purchases of software did not specify the 
application for which the software was being purchased. 
Year-end invoices were submitted for the remaining 
encumbered purchase order balance without the 
documentation necessary to support the invoiced amount. 
Purchase orders and information resource requests did not 
adequately describe deliverables and used the fiscal year-end 
instead of establishing expected delivery dates. 
Invoices frequently did not identify the application upon 
which work had been performed. 
Purchases for software often did not coincide with either the 
start of application development or completion. 
Mavis Georgalis permitted the use of lump sum purchase 
orders and invoices in violation of Contract P0162. 
Lump sum invoices were not supported and sometimes 
charged for services performed outside the purchase order 
time period. 
Mavis Georgalis approved payment for six ARCIS 
applications totaling $1,500,000 ($250,000 x 6 applications). 
Project records and corroborating interviews indicate that 
YEI only implemented three applications. 
In April of 2001, the Department paid maintenance fees for 
six application licenses although only three were in 
production. According to the maintenance agreement, 
maintenance included ARCIS upgrades, however, YEI had 
previously invoiced and been paid for upgrading ARCIS to 
version 7.2. These charges should have been covered by the 
maintenance agreement and reduced the amount charged for 
maintenance. 
The Department paid YEI for services relating to upgrading 
ARCIS version 7.0 to version 7.2 even though the upgrades 
were never implemented. 


CONTRACT P0162 
Our review indicated that Mavis Georgalis did not enforce all 
contract terms, specifically: 


Mavis Georgalis did not issue task assignments describing 
the personnel classifications required for each task. 

YEI did not staff the Central Infrastructure Development 
Facility. Most work was performed at the YEI’s office in 
Oviedo, Florida, which denied the Department the 
opportunity to directly supervise EDMS efforts. Approval of 
the use of personnel outside of Tallahassee was not in 
evidence. 
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Mavis Georgalis did not consistently require monthly status 
reports from YEI. 

Mavis Georgalis issued lump sum purchase orders and 
approved the payment of lump sum invoices that did not 
detail either the hours worked or tasks performed by YEI’s 
employees. 


Our conclusions were based on review of Contract P0162 and the 
Information Systems Development Methodology. 


We also reviewed a variety of literature which provided common 
characteristics of successful project management. Based on this 
research, the following are characteristic of successful project 
management: 


Planning to control the project and ensuring successful 
results. 

Written project objectives that establish calendar date 
deadlines. 

Peer reviews of project manager work, including plans, 
requirements, architecture, design, code, and test cases, are as 
effective or even more effective than testing. 

Measurement of development process success against an 
industry standard such as the Capability Maturity Model 
(CMM) from the Software Engineering Institute at Carnegie 
Mellon University. 

Performance of risk management activities to reduce the 
likelihood of an undesirable outcome and encourage the use 
of processes that identify, assess, classify, plan, track and 
mitigate risks. 

Use of well-defined requirements to establish the project 
scope and track progress by evaluating interim milestones. 
Project tracking utilizing tools such as Gantt Charts. 

A project manager skilled in leadership, management, 
negotiations, problem-solving and communications. 

Project wrap-up/post project summaries that discuss ways to 
improve future projects and assess project success at several 
intervals throughout the project’s lifecycle. 

Use of a best practices library of the methods that have 
worked in previous projects. 


Additionally our conclusions were based on review and analysis of 
the following information: Purchase Orders; 


Invoices; 

Information Resource Requests; 

EDMS Project Schedules; 

Specialized Technology Status Reports; 
YEI Status Reports; and 

YEI Estimates. 
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Cause The following factors contributed to the failures in contract and 
project management: 


The former EDMS Project Manager, Mavis Georgalis, did 
not exert the oversight and guidance necessary for this project 
to be successful; 

Mavis Georgalis did not develop and implement adequate 
procedures, monitor vendor performance, or address 
deficiencies in vendor performance; 

Mavis Georgalis failed to adhere to and enforce contractual 
provisions, Florida Statutes, and Department procedures; 
Mavis Georgalis circumvented controls and failed to 
safeguard Department resources; 

Management did not establish a control structure to ensure 
adequate contract and project management oversight. 

Senior management’s over-reliance on Mavis Georgalis; and 
Weaknesses in the control environment related to training, 
specific contract and project management procedures; and 
transparency and accountability. 


Impact/Risk Failure to adequately perform contract and project management 
duties exposes the Department to the following risks: 


Risk that the Department will not support core values and in 
so doing, will fail to meet Department business objectives; 
Risk that the Department may pay for goods and services not 
received, pay too much for goods and services, and pay for 
goods and services not requested; 

Risk that the project will not be completed in a timely 
manner; 

Risk that the project will not result in the efficient and 
effective implementation of EDMS applications; and 

Risk that the project will not be cost effective. 


Recommendation To strengthen OIS contract and project management processes, we 
recommend that the Assistant Secretary for Finance and 
Administration require: 


The Business System Support Office Manager to develop, 
implement, enforce, and monitor project management 
procedures that includes current project management 
practices and emphasizes the importance of ethics and fiscal 
responsibility. These procedures should require Project 
Managers to: 
o Assess the reasonableness of invoices in relation to 
actual project progress; 
o Review invoices for statutory and procedural 
compliance; and 
o Document that all goods and services received 
conform to requirements and specifications 
established in the contract, base line documents, 
and/or task assignments. This may be accomplished 
by adding a signed statement to the front of the 
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Purchase Order Delivery Notice when submitting an 
invoice for payment. 
= The Chief Information Officer to better support OIS’ project 
managers by assigning an OIS employee, as envisioned by 
the ISDM, to serve as contract manager for Type B 
(Contractual/Miscellaneous) and other atypical, higher-risk 
contracts; 
= The Chief Information Officer to incorporate 
standardized and comprehensive contract and project 
management training into OIS’ current training program. 
This training should emphasize ethics and responsibility 
and include elements of the Department’s Project 
Management Training course that apply to the 
management of Type B (Contractual/Miscellaneous) 
contracts. 
= The Chief Information Officer require Information 
Resource Requests and Purchase Orders to include the 
deliverables to be received, establish the required date of 
completion, hours to be expended, billing rates, and costs 
to be reimbursed when the primary procurement 
document (e.g., contract, task assignment, etc...) does not 
specifically these elements. 
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APPENDIX A - 





Interim Advisory Findings and Recommendations 





(As previously published in the March 31, 2005 Interim Advisory Memorandum 330-4002.) 


Audit Finding 1 
Development of EDMS Initiative - Contract P0162 


OBJECTIVE 1 
Did the project development procedures, specifications, and management direction effectively 
ensure the realization of an electronic document management system? 


CONCLUSION 

The project goals, procedures, specifications, and management direction did not effectively 
ensure the realization of an electronic document management system. We did not assess or draw 
conclusions regarding the overall success of the EDMS initiative. 


SUPPORTING EVIDENCE 

The FSC selected an implementation strategy that was inconsistent with the stated EDMS goals. 
An information technology application implementation methodology was used to effect change 
that required a broader enterprise-wide business process re-engineering methodology. Business 
process re-engineering involves redesigning the way work is done to better support the 
organization’s mission and to reduce costs. Re-engineering identifies, analyzes, and redesigns an 
organization’s core business processes, resulting in decreased costs and improved quality, 
service, and speed. These expected benefits closely parallel the Department’s EDMS goals as 
stated in Policy 000-325-008-a. 


The FSC strategy presumes that a significant change in business processes and Department 
culture would happen organically (effortlessly) — this was not a reasonable expectation. The 
following items are symptomatic of not providing adequate direction and oversight: 


e The ESC did not plan or direct the initiative in a manner conducive to meeting the stated 
goals of the EDMS project. 

e The FSC did not develop an implementation plan that included critical elements and 
milestones and specific time tables. 

e The Functional Specification Document, designated as the guide and five-year 
implementation plan, emphasized application development and did not address the 
broader business process goals. 

e EDMS requirements established in the RFP and ITN contained functional specifications 
for features desired by the Department but not readily available in the marketplace. 

e The FSC and ESC dissolved in early 1998 with the appointment of the EDMS Project 
Manager. After early 1998, there was little involvement on the part of senior 
management. 

e As the OIS Specialized Technologies Manager, the EDMS Project Manager was 
responsible for both GIS and EDMS. 

e The EDMS Project Team lacked a clear team charter and project plan. Furthermore, the 
EDMS Policy applicable during the development phase of the initiative defined duties for 
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the Project Team which were inconsistent with the achievement of the broader EDMS 
goals. See Appendix A for Policy 000-325-008-a, Electronic Document Management. 

e The Project Team did not consist of individuals who were qualified and trained to carry 
out re-engineering projects and did not represent all of the functional disciplines affected 
by the project implementation. 

e The criteria used to select the EDMS Project Manager was inconsistent with the stated 
goals of the EDMS project. The key qualifications for the Project Manager focused only 
on experience with electronic document management, rather than on experience with 
implementing enterprise-wide business process re-engineering initiatives. The EDMS 
Project Manager did not posses the experience required to successfully bring about the 
Department-wide business process changes. 

e The EDMS vendor integrator, YEI, did not possess the software expertise or demonstrate 
the requisite experience to successfully implement an EDMS. More information 
regarding this issue is included in Finding 2. 

e Management did not to publish EDMS Procedure 325-055-015. 


CRITERIA 
Policy 000-325-008, Electronic Document Management 


Policy 001-325-008, Electronic Document Management and Geographical Information 
Systems. 


Functional Specifications Document 

Policy 000-325-008 established the Functional Specifications Document as the Department’s 
five-year guide and EDMS implementation plan. This document recommended that the 
Department perform a broad based re-engineering study to analyze current business practices and 
to determine potential re-engineering costs and benefits. The Functional Specifications 
Document also recommended that the Department perform a re-staffing study to address the 
impact of personnel changes resulting from business process re-engineering. 


Best Practice Research 
The Department of Defense’s Business Process Re-engineering Guide, the U.S. Government 
Accountability Office’s Business Process Re-engineering Assessment Guide, and a variety of 
other literature provided common characteristics of successful business process re-engineering 
projects. This research indicates that the following are necessary to ensure the success of the 
EDMS initiative: 
e Ongoing involvement of senior management; 
e Clear project guidance and direction; 
e Well developed, clearly communicated objectives which focus on strategic business 
needs; 
e Experience in business process re-engineering; 
e Development and use of a comprehensive implementation plan; 
e A charter outlining the scope and goals of the project, the team’s authority and 
interactions with executive management, and any other special considerations; and 
e A project plan that describes the activities, deliverables, and time frames for the effort 
and serves as the baseline for managing activities and measuring progress. 
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The Department of Defense’s Business Process Re-engineering Guide also provided key 
concepts to consider when planning and executing a business process re-engineering effort, such 
as: 
e Minimize “as-is” business process analysis; 
e Streamline business operations; 
e Hone in on the real business requirements and challenge processes that merely 
accommodate the way it has always been done; 
e Reduce the cost of doing business by eliminating: 
o Obsolete and inefficient processes; 
o Obsolete regulations and controls; 
o Lengthy review and approval cycles; and 
e Ensure business processes are integrated across all impacted functional areas. 


CAUSE 
During the development of the EDMS initiative, management did not: 
e Develop an implementation methodology that was consistent with the stated goals of the 
EDMS; 
e Provide OIS sufficient direction and oversight; 
e Establish and communicate specific EDMS goals, procedures, deliverables, milestones, 
or roles and responsibilities that efficiently and effectively ensured the realization of an 
electronic document management system. 


IMPACT/RISK 
Not effectively developing the EDMS initiative has resulted in the Department inefficiently 
expending resources. 


RECOMMENDATION 

In response to issues affecting EDMS, management updated EDMS Policy Statement 
000-325-008-a, deleting business process re-engineering references and better aligning 
EDMS Policy with actual project goals. In order to provide better oversight and 
direction, management re-assigned the responsibility for the EDMS and GIS projects to 
the Business System Support Office. 


However, persistent obstacles have impeded the efficient and effective implementation of 
EDMS. In order to overcome these obstacles and to provide management with a means 
of measuring EDMS successes, we recommend the Secretary task the Chief Information 
Officer with: 
1) Establishing specific project goals, deliverables, and milestones to allow the Information 
Resource Management Council to monitor the progress of the EDMS initiative. 


2) Creating EDMS procedures that contain specific operating requirements and instructions 
to implement Policy Statement 001-325-008-c. 


MANAGEMENT’S RESPONSE 


We concur with the recommendation and would offer that the recommendation is currently being 
addressed. It is management’s position that the current EDMS policy (Topic No.: 001-325-008- 
c) properly reflects the desired direction, at a policy level, the department intends to pursue for 
EDMS. We would further offer that the EDMS application projects that are currently underway 
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within the Office of Information Systems have properly stated goals, deliverables and milestones 
in accordance with the OIS Information Systems Development Methodology. As is the case 
with any computer application development activity, including EDMS, business process re- 
engineering typically occurs based on users’ review of their business processes and decisions 
made through joint application development sessions. The involvement of OIS serves as a 
catalyst for making BPR happen, but in reality BPR becomes a by-product of several 
independent activities. Computer application development and technology deployment is just 
part of many components that enable BPR. 
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Audit Finding 2 
Vendor Selection Process 


OBJECTIVE 
Did the vendor selection process for the EDMS vendor integrator fully comply with established 
procedures and efficiently and effectively ensure an appropriate vendor selection? 


CONCLUSION 
The vendor selection process did not fully comply with established procedures and did not 
effectively ensure an appropriate vendor selection. 


SUPPORTING EVIDENCE 
While our overall conclusion is that the vendor selection process did not effectively ensure an 
appropriate vendor selection, the process was well executed in a number of respects: 


The vendor section process conformed in all material respects to applicable time and 
resource constraints; 

The Technical Evaluation Team took an organized approach to evaluating the vendor 
proposals; 

The Live Test Demonstration process involved a wide cross-section of Department 
personnel representing diverse areas of expertise; 

USI compiled vendor reference information during the evaluation phase; and 

The ITN process required the completion of a pilot project prior to execution of the 
contract. 


However, the following conditions are the basis for our conclusion that the vendor selection 
process did not effectively ensure an appropriate vendor selection: 


In their January 1998 proposal, YEI misrepresented their EDMS experience. The 
proposal included references for Siemens-Nixdorf installations of ARCIS software on 
which YEI had not participated. YEI entered into a partnership agreement with Siemens- 
Nixdorf in June 1998. 

YEI had not previously installed production ARCIS on an enterprise-wide basis and 
further had limited practical experience with electronic document management 
implementation. 

Under Ms. Georgalis’ project management in District 2, YEI failed to deliver software 
under ICF Kaiser Contract C5767. This information was not disclosed to the other 
members of the Technical Evaluation Team. 

As the EDMS Project Manager, Ms. Georgalis did not disclose critical first-hand 
experience with YEI and did not heed USI warnings regarding deficiencies in YEI’s 
practical electronic document management experience. The following quote indicated 
the nature of these warnings: 


“Yang showed no knowledge of Document Management. They demonstrated 

very little knowledge of the Siemens product suite. The only product knowledge 

that was displayed was learned during the week before the live test 

demonstration. They demonstrated no ability to implement and maintain the 

system being proposed.” 

USI’s Yang/Siemens Live Test Demonstration Report dated May 20, 1998 
References were not adequately considered by the Technical Evaluation Team, which 
resulted from the EDMS Project Manager withholding pertinent vendor performance 


information. The Project Manager awarded YEI 15 out of 20 points for “references,” 
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tying the highest score Ms. Georgalis awarded in this category despite the fact that she 
was aware of YEI’s prior performance issues. 

e All phases of the vendor selection process were not performed by at least three 
individuals as required by Section 287.057, F.S., and Department Procedure 375-040- 
020. 

e YEI failed to satisfactorily complete the Live Test Demonstration project and was 
unsuccessful in demonstrating the ARCIS software. 

e Live Test Demonstration results were not adequately considered. Source documentation 
was not available to verify demonstration results. Neither a comprehensive summary nor 
a formal statement of Live Test Demonstration results was in the procurement file or 
other vendor selection records. 

e The two key criteria used by the EDMS Project Manager, to support the selection of YEI, 
were their proposed decentralized solution and compatibility with the Department’s 
current operating system (Windows NT Version 4.0). These two criteria were 
inappropriately emphasized over the primary selection criteria — the results of the 
technical proposal evaluations, Live Test Demonstrations and vendor references. 

e YEI staffed the pilot project with one on-site individual hired after the Live Test 
Demonstration. This individual had no experience with ARCIS or the Department’s 
requirements. Ms. Georgalis directed other Department consultants to assist YEI staff 
with installation of ARCIS software in an attempt to implement the CDMS pilot project. 

e YEI failed to successfully implement a pilot project prior to contract execution as 
required by the ITN. Implementation of a Construction Document Management System 
(CDMS) pilot project was required by the ITN as a condition of executing the contract. 

e ITN requirements were further violated in that no résumés, Price Proposal scoring, or 
formal statement of vendor selection results were in the procurement file or other vendor 
selection records. 


CRITERIA 

Section 287.057, Florida Statutes, Procurement of Commodities and Contractual Services 
This statute contains procurement requirements, including the requirement that a selection team 
consist of at least three employees who have experience and knowledge in the program areas and 
service requirements for which contractual services are sought. 


Department of Management Services Memorandum #21 
This memorandum provides ITN process guidelines, including basic methodologies for 
competitive negotiations. 


Procurement Procedure 375-040-020, Procurement of Commodities and Contractual 
Services 

This procedure requires the Contractual Services Unit to maintain a contract file which contains 
all pertinent documentation of the procurement, contracting, and the administrative phases. It 
also requires that the vendor selection process complies with Section 287.057, F.S., and that 
three or more people perform the selection. This procedure also describes the procurement 
process as the preliminary portion of the overall acquisition process, which includes the 
development of the scope of services, the preliminary development of payment methods/terms 
and conditions, the selection of the contractor, and the establishment of pricing information. 


ITN Document JA0698D1, Sections 6.0, 7.1, and 7.2 Vendor Background and Evaluation. 
This document: 
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e Requires the vendor to describe their installed base of the software product being 
proposed, which must include the number of installations, configuration, architecture, 
seats, and main customers utilizing the products; 

e Establishes the evaluation and ranking criteria for each phase of the ITN process; 

e Requires specific criteria and forms for developing price proposals; 

e Requires vendors to provide résumés of the personnel who may be used to provide 
professional services; and 

e States that the selected vendor will develop a pilot application that must pass a quality 
assurance test. 


1997-1998 Sterling Quality Challenge 

In its response to the Sterling Quality Challenge Self Assessment Report, the Department stated 
that senior leaders promote the values of hard work, quality, teamwork, honesty, loyalty, and 
accountability. The Department further indicated that senior leaders expect employees to adhere 
to the Agency’s values, do the job right the first time, work in an efficient manner, and provide 
quality customer service. 


Management Best Practices 

The Department has adopted five fundamental leadership competencies necessary for excellence. 
These competencies emphasize the ability to develop and implement strategies to maximize 
employee performance excellence and foster high ethical standards in meeting the Department’ s 
vision, mission, and goals. They also require management to develop and implement an 
organizational vision and to incorporate that vision into the State’s Transportation goals and the 
Department’s core values. The core competencies require management to demonstrate business 
judgment and be able to manage and apply financial, human resources, and technology 
management to accomplish the organization’s missions. Effective and efficient decision making 
epitomizes sound business judgment. Results driven management stresses accountability and 
continuous improvement. 


Department Core Values 
Integrity — We are committed to honesty, loyalty, and a high standard of ethical conduct. 


Respect — We value diversity, talent, and ideas. We believe every individual should 
contribute and have the opportunity to be heard. 


Excellence — We achieve performance excellence through hard work, innovation, 
creativity, and prudent risk taking. 


Team Work — We accomplish more by working together and relying on each other. 
CAUSE 
The following factors contributed to the vendor selection process failure: 


e The EDMS Project Manager withheld pertinent information and did not ensure 
compliance with all statutory and procedural requirements. 
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The Department has not trained or developed all Project Managers to be effective 
Contract Managers. Effective contract management skills are necessary due to increased 
outsourcing and privatization. 


IMPACT/RISK 
An inappropriate vendor selection increases the risk that the EDMS initiative will not be 
successful and agency resources will be wasted. 


RECOMMENDATION 

To address the systemic issues impacting both the EDMS initiative and the Department as a 
whole, we recommend the Secretary task the Executive Board to establish and maintain an 
organizational culture that balances project completion needs with compliance, transparency, and 
accountability. To create this balance, the Executive Board should establish the expectations 


that: 


Proper stewardship of State resources is a fundamental responsibility of Department 
managers and staff; 

Department employees must ensure that resources are used efficiently and effectively to 
achieve intended results; and 

Resource use must be consistent with the Department’s mission, in compliance with law 
and regulation, and minimize the potential for waste, fraud, and mismanagement. 


To establish the above expectations, the Executive Board should: 


1) 


2) 


3) 


4) 


5) 


Evaluate current Department training for coverage of matters of ethics and governance 
and enhance project and contract manager training. 


Ensure that Department project and contract managers are adequately trained in matters 
of fiscal responsibility. We recommend all Department project managers be required to 
attend the Department of Financial Services’ course “Getting What You Pay For’, or a 
comparable course. 


Establish a position responsible for contract management in major functional areas to 
support project managers. These individuals must possess the appropriate contract 
management skill sets and have an ongoing program of contract management training. 


Establish and require adherence to the Project Management Handbook. 
Implement, enforce, and monitor adequate procurement processes and procedures, which 
require a detailed justification of the selection, thorough reference checks, including the 


vendor’s Department experience, and require vendor selection committees to sign 
disclosure statements for actual or perceived conflicts of interest. 
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MANAGEMENT’S RESPONSE 
Management concurs with the conclusion, but we take issue with the recommendation. 


We certainly recognize that this project had a number of problems during the initial procurement 
stages and suffered from inappropriate contract management. However, we do not feel that this 
particular circumstance merits the broad, sweeping changes noted in the recommendation. The 
advisory memorandum is focused on the specifics regarding the management of the EDMS 
initiative. We fully understand the problematic issues surrounding the early phases of this 
project and believe we have implemented measures to correct those issues for the project. To 
offer a recommendation as stated, we feel the OIG would need to issue an advisory report that 
looks at the department as a whole, not just one specific project. 


OIG RESPONSE 


We believe the recommendations lay out a prudent course of action supported by previous OIG 
audits, as well as a multi-agency state audit performed by the Governor’s Chief Inspector 
General. The Chief Inspector General’s audit report, titled ‘Road Map to Excellence in 
Contracting,” documents that contracting issues relating to compliance, ethics, accountability, 
and stewardship are systemic, not for any one agency, but for all State agencies. 


While we believe that the issues surrounding P0162 are indicative of a systemic problem 


impacting the Department as a whole, we are receptive to looking at the issue from a broader 
perspective as suggested... We will work with management to accomplish this task. 
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APPENDIX B - Audit Team and Statement of Accordance 





This audit was conducted by Karen Taylor, supervised by Carlos Mistry, and directed by 
Joseph Maleszewski. 


Statement of Accordance - Sections 20.23 and 20.055, Florida Statutes, require the 
Florida Department of Transportation’s Inspector General to review, evaluate and report 
on policies, plans, procedures, accounting, financial and other operations of the 
Department and to recommend improvements. This audit was conducted to assist 
Department management in the review of reliability and validity of selected key 
performance measures. 


This engagement was performed in accordance with applicable Government Auditing 


Standards issued by the Comptroller General of the United States and Standards for the 
Professional Practice of Internal Auditing published by the Institute of Internal Auditors. 
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APPENDIX C - Distribution List 
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